Frequency-based projection segmentation

ABSTRACT

A method for segmenting a projected pattern in an image recorded by a camera includes recording, by a camera in a learning phase, a multiplicity of images produced by virtue of a light source projecting the pattern from a plurality of different angles onto a projection surface in a clean room, wherein the projection surface has a plurality of respectively different distances from the light source for each angle; transforming the multiplicity of images into a frequency domain representation; obtaining a value range of occurring frequencies from the frequency domain representation of the multiplicity of images; and masking, in an application phase, frequencies other than the frequencies lying in the value range in a frequency domain representation of the image recorded by the camera, wherein a difference image produced in this manner is transformed back from the frequency domain representation.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims benefit to German Patent Application No. DE 102017 105 910.5, filed Mar. 20, 2017, which is hereby incorporated byreference herein.

FIELD

The present invention relates to a method and a system for segmentingimage content containing a projected pattern.

BACKGROUND

A triangulation, i.e. ranging by accurate determining of an angle withina triangle formed in space by two points of the optical system and arespective point of an object, is a standard method used in the 3Dreconstruction of a space situated in front of an optical system. Thetriangulation can be carried out passively or actively. Conventionally,the passive triangulation is known, in which respectively one observer,for example a camera, is situated at the two points of the opticalsystem. By way of example, this principle is realized in a stereo camerawhich records an object in space from different angles. However, theactive triangulation, in which one observation point is replaced by alight source, i.e. the triangle consists of a light beam, irradiatedobject and camera, is the measurement principle most commonly used inoptical ranging. Here, the ranging may also take place along a visualplane if a light stripe is projected in place of a light beam. A profileof the light stripe imaged in the camera can be converted into adistance profile along the light stripe by means of triangulation. Then,the fast measurement of a complete space is obtained by a structuredlight projection, in which different stripe patterns, for example with astripe thickness that doubles in each recorded image of the sequence,are projected onto the object in quick temporal succession. Thus, the 3Dreconstruction of the space can be calculated in real time from theknown set of the stripe patterns and the images thereof in the camera.Here, the quality of the 3D reconstruction depends strongly on thecorrect identification of illuminated and non-illuminated areas.

3D reconstruction occurs in various technical applications, for examplein a contactless measurement of objects, in medicine and dentistry, butespecially in industry when controlling the form of workpieces or whendesigning a new form for a product. In real time, it is of essentialimportance to autonomously moving systems, the surrounding space ofwhich needs to be explored. It is likewise used in a driver assistancesystem for assisting a driver of a motor vehicle and it forms the basisfor a computer-controlled image analysis.

In a motor vehicle, the light source advantageously already is providedby a headlamp which, together with a camera installed in the frontregion, forms the optical system. Here, accurate knowledge about thevehicle-specific light distribution of the headlamp is of greatimportance for the quality of the image processing, making a calibrationof a headlamp-camera system indispensable. By way of example, thedocument DE 10 2011 109 440 A1 describes a method which can be used toadjust and/or calibrate a headlamp of a vehicle.

By means of the headlamp, it is possible to project a pattern onto avehicle near field, said pattern being used for image processing withinan image recorded by the camera. Depending on a surface structure in thevehicle near field or on objects situated therein, the projected patternis recorded in deformed fashion by the camera. A certain region which isof particular interest for a further evaluation may be stipulated forthe image recorded by the camera. Such a region is commonly referred toas “region of interest”, also abbreviated as ROI in technical jargon.

According to the prior art, a feature search is carried out within thestatically predetermined ROI during the image processing and at leastone feature is extracted. Here, a feature consists of an imaging of apattern projected by the headlamp which should represent a particularlygood characteristic in the image. By way of example, such a pattern mayconsist of a checkerboard pattern. It should still be uniquelyidentifiable, even in the case of a strong deformation. The term“feature” is also used for this in technical jargon.

By way of example, the feature search may contain an edge detection.Various algorithms are known to this end. The Canny algorithm isspecified here as an example; it supplies an image only still containingedges in the ideal case after carrying out various convolutionoperations.

The position of the found feature in the image of the camera and ageometric data record from the projected form of the headlamp form aso-called feature pair. The 3D reconstruction of the space situated infront of the headlamp-camera system is effectuated on the basis of thefound feature pairs by means of triangulation; said 3D reconstructionmay also consist of a pure depth map.

Identifying an erroneous feature occurs in the aforementioned featuresearch, the origin of said feature not lying in the projection of apattern by the headlamp but said erroneous feature nevertheless beingassigned a geometric data record from the headlamp and forming anerroneous feature pair therewith. What may occur particularly in thecase of vehicle operation is that image components which do not containa projected pattern repeatedly occur in the statically predeterminedROI, with the feature search however leading to erroneous feature pairsin said image component. A triangulation based on such an erroneousfeature pair thus leads to an erroneous depth map.

In order to solve a problem of the static ROI—that it may contain imagecomponents without projected patterns—the prior art has proposed anadaptive restriction of the ROI to the projected pattern over the courseof time. To this end, a light/dark boundary, abbreviated HDG in theGerman technical jargon, is extracted for each image recorded by thecamera between the comparatively light projected pattern and thecomparatively dark vehicle near field, and the ROI is adaptivelysegmented in this respect. The further image processing then only occursin the continuously adapted ROI. However, a disadvantage of thisprocedure is that objects may appear in the regions in the vehicle nearfield predetermined by the light/dark boundary, said objects having acertain brightness but not originating from the projected pattern andthen having erroneous feature pairs as a consequence.

SUMMARY

In an embodiment, the present invention provides a method for segmentinga projected pattern in an image recorded by a camera. The methodincludes recording, by a camera in a learning phase, a multiplicity ofimages produced by virtue of a light source projecting the pattern froma plurality of different angles onto a projection surface in a cleanroom, wherein the projection surface has a plurality of respectivelydifferent distances from the light source for each angle; transformingthe multiplicity of images into a frequency domain representation;obtaining a value range of occurring frequencies from the frequencydomain representation of the multiplicity of images; and masking, in anapplication phase, frequencies other than the frequencies lying in thevalue range in a frequency domain representation of the image recordedby the camera, wherein a difference image produced in this manner istransformed back from the frequency domain representation.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described in even greater detail belowbased on the exemplary figures. The invention is not limited to theexemplary embodiments. All features described and/or illustrated hereincan be used alone or combined in different combinations in embodimentsof the invention. The features and advantages of various embodiments ofthe present invention will become apparent by reading the followingdetailed description with reference to the attached drawings whichillustrate the following:

FIG. 1 shows images, recorded by a camera, of checkerboard patternsprojected into a vehicle near field from different angles and distances,and the frequency domain representation thereof according to a method ofan embodiment of the invention;

FIG. 2 shows a schematic illustration of a multiplicity of images of alearning phase and the superposition thereof in the frequency domainrepresentation according to a method of an embodiment of the invention;

FIG. 3 shows a schematic illustration of steps of a method according toan embodiment of the invention for segmenting the projected pattern;

FIG. 4 shows a difference image in a frequency domain representation anda selection of Gabor wavelets according to a method of an embodiment ofthe invention;

FIG. 5 contrasts various methods from the prior art for segmentingprojected checkerboard pattern; and

FIG. 6 schematically shows the relationship between the selection of aGabor wavelet and the resultant projected checkerboard pattern accordingto a method of an embodiment of the invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide methods and systems for asegmentation of an image recorded by a camera which, for the furtherimage processing, only uses those regions which originate from a patternprojected into a vehicle near field.

According to an embodiment of the invention, a method for segmenting aprojected pattern in an image recorded by a camera is presented.Initially, in a learning phase, a multiplicity of images recorded by thecamera are produced by virtue of a light source projecting the patternfrom a plurality of different angles onto a projection surface in aclean room and the projection surface having a plurality of respectivelydifferent distances from the light source for each angle. Here, the roomis only prepared as a clean room for recording these images, with nointerfering objects being situated therein. Each image of themultiplicity of images is transformed into a frequency domainrepresentation, with a value range of occurring frequencies beingobtained by superposing all frequency domain representations of themultiplicity of images. In the application phase, other frequencies thanthe frequencies lying in this value range are masked in a frequencydomain representation of the image recorded by the camera, a differenceimage produced in this manner is transformed back from the frequencydomain representation, and said difference image then is made availablefor further image processing.

In a possible configuration, the multiplicity of images obtained in thelearning phase form a ground truth, from which a frequency corridor isdetermined. In technical jargon, an image information item or, equally,the frequency domain representation thereof, which, like in theaforementioned learning phase for example, only contains the purepattern projected without further influences or aberrations, is referredto as ground truth. Then, in a two-dimensional frequency domainrepresentation, at least the range between a lowest frequency and ahighest frequency of the ground truth in each of the two dimensionsdefines a frequency corridor, outside of which other frequenciesoccurring in the application phase are masked. In order to allow somesmall tolerance in this case, the frequency corridor may also beslightly enlarged for safety purposes. Advantageously, a headlamp isused as a light source in the case of a motor vehicle.

The transformation of an image into the frequency domain can be carriedout by means of a Fast Fourier Transform. This is particularlyadvantageous if the image processing should be effectuated in real timein the application phase. In the same way, the use of an inverse FastFourier Transform is advantageous during the back transformation fromthe frequency domain.

In the learning phase, it is advantageous to select the angles anddistances used in the production of the multiplicity of images with asystematic increment between a respectively smallest value and arespectively largest value. Here, for each selected distance,respectively one image of the projected pattern is taken by the camerafor each angle.

In a possible configuration of a method according to the invention, thefrequency domain representations of the images recorded by the camera inthe learning phase are individually stored with the respective value forangle and distance, wherein a reference map is created, said referencemap uniquely linking the frequency domain representation of therespective image to the angle and distance of the projected pattern.

Advantageously, a checkerboard pattern is selected for the pattern to beprojected. Uniformly alternating bright and dark area elementscorrespond to characteristic frequencies in the frequency domainrepresentation, the values of said characteristic frequencies lying everhigher in the frequency domain representation the smaller the areaelements appear in, for example, the vehicle near field. Hence, it ispossible to assign to the characteristic frequencies a distance at whichthe projected pattern appears in the image recorded by the camera.

In a configuration of a method according to the invention, otherfrequencies than the frequencies lying in the value range based on theground truth are masked from the frequency domain representation of theimage recorded by the camera in the application phase, and angles andfrequencies corresponding to the reference map are determined. Forfurther image processing, it is advantageous to use Gabor wavelets as abasis for an image processing kernel, the orientation and frequency ofsaid Gabor wavelets within the frequency corridor corresponding to thevalues for the found angles and frequencies from the reference map.Then, the segmentation of the projected pattern in the image recorded bythe camera in the application phase is carried out using this imageprocessing kernel.

FIG. 1 shows images 110, 120, 130, recorded by a camera, of acheckerboard pattern 112, 122, 132 projected into a clean room fromdifferent angles and distances, and the frequency domain representation114, 124, 134 of said images. The distance of the projected checkerboardpattern 112 is 2.5 m in image 110. The distance is 5.5 m in image 120and the projected checkerboard pattern additionally is rotated by 5°. Inimage 130, the checkerboard pattern is projected without rotation from adistance of 5.5 m. Additionally, the image 130 has a rectangularfalsification 136. All three images 110, 120, 130 have been convertedinto their frequency domain representation 114, 124, 134 by way of aFast Fourier Transform 102. What all three frequency domainrepresentations 114, 124, 134 have in common are the brightlyillustrated frequencies 104 and 106, which alone correspond to theprojected checkerboard pattern in a spatial domain of the images 110,120, 130. Further visible frequencies 116 are artifacts. The rectangularfalsification 136 leads to a horizontal and vertical stringing togetherof frequencies 138 and 139 in the frequency domain representation 134.

FIG. 2 schematically shows the generation of a multiplicity of images202 in the learning phase, in which a checkerboard pattern is projectedinto a clean room. Each individual one of the images 202 has beenrecorded for a different angle and/or distance of the projected pattern.The individual images 202 are converted into a frequency domainrepresentation 204. A value range 212 of occurrent frequencies becomesvisible in 210 from the superposition 208 of all frequency domainrepresentations 204, as a result of which it is possible to set afrequency corridor. Moreover, the frequency domain representations 204form a data basis 206 for a reference map which facilitates theassignment of frequencies occurring in a frequency domain representationto an angle and distance of the projected checkerboard pattern.

FIG. 3 shows the processes during the application phase when carryingout an embodiment of a method according to the invention. By way ofexample, a real image 302 recorded by the camera also contains unwantedimage components 320 in addition to the projected checkerboard pattern322. After a transformation 312 into a frequency domain, the frequencies332 caused by unwanted image components 320 are masked by applying 314to the frequency domain representation 306 of the recorded image 302 thefrequency corridors 330 obtained from the multiplicity of images 328during the learning phase 304 and a difference image 308 is obtained byway of this process 316. The difference image 308 only still containsthe frequencies 334 which, after a back transformation 318 into thespatial domain, lead to the projected checkerboard pattern 336.Schematically, this means that the method according to the inventionblocks the unwanted image components, as indicated by the arrow 326, butsegments the projected checkerboard pattern for further imageprocessing, as indicated by arrow 324. Moreover, using the referencemap, an angle and a distance of the projected checkerboard pattern canbe assigned for the further image processing by way of the frequencies334 ascertained in the difference image 308.

At the top, FIG. 4 shows a difference image 402 which arose from thefrequency domain representation of a real image recorded by the camera.An angle and a distance of the projected checkerboard pattern areassigned by means of the reference map to the frequencies 404established in the frequency corridors 406 of the ground truth. By wayof example, this results in values of 180+/−22.5 degrees for the angleand 0.02+/−0.005 for the frequency, wherein the variation about the meanvalue corresponds to a safety corridor. Specifying a frequency is anobvious representation in the frequency domain for a checkerboardpattern and this is advantageously used here in place of a distance asthis yields a suitable selection of Gabor wavelets 411, 412, 413, 421,422, 423, 431, 432, 433 for the further image processing. Image 422shows a two-dimensional Gabor wavelet with an orientation anglecorresponding to the aforementioned mean values of 25 degrees and awavelength with a value of 7. According to the aforementioned safetycorridors, this leads to the emergence of the images 411, 412, 413 withan orientation angle of 55 degrees, the images 421, 422, 423 with anorientation angle of 25 degrees and the images 431, 432, 433 with anorientation angle of 5 degrees, and also of the images 411, 421, 431with a wavelength with a value of 5, the images 412, 422, 432 with awavelength with a value of 7 and finally the images 413, 423, 433 with awavelength with a value of 9.

FIG. 5 compares various methods from the prior art for segmenting theprojected checkerboard pattern. The starting point is a test image 502which, in addition to the projected checkerboard pattern 504, also hasother image components, for example a checkerboard pattern 506 paintedin the vehicle near field and a bright horizontal and vertical lightstripe. If the test image 502 now is subject to simple edge detection510, for example using the Canny edge detector, the result 511, inaddition to the correctly identified projected checkerboard pattern 514,also would contain the painted checkerboard pattern 516 and a number offurther unwanted artifacts 518. If the edge detection 520 is restrictedto the ROI, for example by restricting the ROI to regions 521 defined bymeans of the light/dark boundary, the result 522 could nevertheless showimage components with artifacts 518 in addition to the projectedcheckerboard pattern. Only the adaptive determination 530 of the ROI, inwhich, in 531, regions within a light/dark boundary are initiallyidentified and then, in 532, the ROI 536 is restricted to the projectedcheckerboard pattern while dark image components 534 are ignored,produces the desired result 538 in 533.

FIG. 6 schematically shows the relationship between the selection of aGabor wavelet and the resulting projected checkerboard pattern. If thetest image 602 were to be imaged by means of a respective Gabor wavelet611, 612, 613 by way of image processing 610, the result 620 would be animage of the projected checkerboard pattern, as illustrated in images621, 622, 623. In detail, a wavelength of 5 with an orientation angle of55 degrees was selected for the Gabor wavelet 611, a wavelength of 7with an orientation angle of 25 degrees was selected for the Gaborwavelet 612 and a wavelength of 9 with an orientation angle of 5 degreeswas selected for the Gabor wavelet 613. The angle and the distance ofthe projected checkerboard pattern in the image emerges accordingly.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive. Itwill be understood that changes and modifications may be made by thoseof ordinary skill within the scope of the following claims. Inparticular, the present invention covers further embodiments with anycombination of features from different embodiments described above andbelow.

The terms used in the claims should be construed to have the broadestreasonable interpretation consistent with the foregoing description. Forexample, the use of the article “a” or “the” in introducing an elementshould not be interpreted as being exclusive of a plurality of elements.Likewise, the recitation of “or” should be interpreted as beinginclusive, such that the recitation of “A or B” is not exclusive of “Aand B,” unless it is clear from the context or the foregoing descriptionthat only one of A and B is intended. Further, the recitation of “atleast one of A, B and C” should be interpreted as one or more of a groupof elements consisting of A, B and C, and should not be interpreted asrequiring at least one of each of the listed elements A, B and C,regardless of whether A, B and C are related as categories or otherwise.Moreover, the recitation of “A, B and/or C” or “at least one of A, B orC” should be interpreted as including any singular entity from thelisted elements, e.g., A, any subset from the listed elements, e.g., Aand B, or the entire list of elements A, B and C.

What is claimed is:
 1. A method for segmenting a projected pattern in animage recorded by a camera, the method comprising: recording, by acamera in a learning phase, a multiplicity of images produced by virtueof a light source projecting the pattern from a plurality of differentangles onto a projection surface in a clean room, wherein the projectionsurface has a plurality of respectively different distances from thelight source for each angle, transforming the multiplicity of imagesinto a frequency domain representation, obtaining a value range ofoccurring frequencies from the frequency domain representation of themultiplicity of images, and masking, in an application phase,frequencies other than the occurring frequencies lying in the valuerange in a frequency domain representation of the image recorded by thecamera, wherein a difference image produced in this manner istransformed back from the frequency domain representation.
 2. The methodas claimed in claim 1, wherein the multiplicity of images obtained inthe learning phase form a ground truth and a frequency corridor isdetermined therefrom.
 3. The method as claimed in claim 1, wherein aheadlamp of a motor vehicle is selected as a light source.
 4. The methodas claimed in claim 1, wherein transforming the multiplicity of imagesinto the frequency domain representation is carried out using a FastFourier Transform.
 5. The method as claimed in claim 1, wherein a backtransformation from the frequency domain is carried out using an inverseFast Fourier Transform.
 6. The method as claimed in claim 1, whereinangles and distances used to produce the multiplicity of images areselected with a systematic increment between a respectively smallestvalue and a respectively largest value.
 7. The method as claimed inclaim 1, wherein a reference map is created in the learning phase withaid of the frequency domain representation of the multiplicity ofimages, wherein the reference map assigns the frequency domainrepresentation of the image to an angle and distance of the projectedpattern.
 8. The method as claimed in claim 1, wherein a distance of theprojected pattern, which consists of a checkerboard pattern, correspondsto a frequency in the frequency domain.
 9. The method as claimed inclaim 7, wherein, using a reference map, at least one angle with atleast one frequency is determined from the frequency domainrepresentation of the image recorded by the camera in the applicationphase, frequencies other than the occurring frequencies lying in thevalue range having been masked in the image, a suitable set of Gaborwavelets for further image processing being selected based on the atleast one angle with the at least one frequency.